Field of the Invention
This invention relates to the fields of confocal and non-confocal imaging of large microscope specimens with particular emphasis on fluorescence and photoluminescence imaging systems, including multi-photon fluorescence, spectrally-resolved fluorescence, and second and third harmonic imaging. Applications include imaging tissue specimens, genetic microarrays, protein arrays, tissue arrays, cells and cell populations, biochips, arrays of biomolecules, detection of nanoparticles, photoluminescence imaging of semiconductor materials and devices, and many others.
Description of the Prior Art
FIG. 1 shows one embodiment of a prior art confocal scanning laser macroscope, as described in U.S. Pat. No. 5,760,951. In this embodiment, the incoming collimated laser beam 102 from laser 100 passes through a beam expander (comprised of lens 104 and lens 106), and is expanded to match the diameter of entrance pupil 112 of laser scan lens 114 (note—entrance pupil 112 as indicated on FIG. 1 simply indicates the position of the entrance pupil. A real stop is not placed at this position). Scanning mirror 110 deflects the beam in the X direction. Laser scan lens 114 focuses the beam to focused spot 116 on specimen 118, mounted on microscope slide 120, and light reflected from or emitted by the specimen is collected by laser scan lens 114, descanned by scanning mirror 110, and partially reflected by beamsplitter 108 into a confocal detection arm comprised of laser rejection filter 130, lens 132, pinhole 134, and detector 136. Detector 136 is located behind pinhole 134. Light reflected back from focused spot 116 on specimen 118 passes through pinhole 134 and is detected, but light from any other point in the specimen runs into the edges of the pinhole and is not detected. The scan mirror is computer-controlled to raster the focused spot across the specimen. At the same time, microscope slide 120, which is mounted on a computer-controlled, motor-driven scanning stage 122, moves slowly in the Y direction. The combination of rapid beam scanning across the specimen while it is moved slowly in the perpendicular Y direction results in a raster-scan motion of focused-laser spot 116 across specimen 118. A computer, represented by computer screen 140, is connected to detector 136 to store and display a signal from detector 136. The computer provides means for acquiring, manipulating, displaying and storing the signal from the detector. This confocal macroscope has properties similar to those of a confocal scanning laser microscope, except that the field of view of the microscope is much smaller.
The instrument shown in FIG. 1 has the ability to adjust the gain of the detector depending on the fluorescence intensity of the fluorophore, and a high-speed preview scan can be used to predict the exposure required for each fluorophore before scanning the final high-resolution image (see PCT application WO 2009/137935 A1). Because the laser scan lens has a wide field of view, large specimens can be scanned in a few wide strips, making it possible to scan very large specimens (up to 6×8 inches in size in one version of a commercial instrument).
Several other technologies are used for fluorescence imaging of large specimens. With tiling microscopes, the image of a small area of the specimen is recorded with a digital camera (usually a CCD camera), the specimen is moved with a computer-controlled microscope stage to image an adjacent area, an image of the adjacent area is recorded, the stage is moved again to the next area, and so on until a number of image tiles have been recorded that together cover the whole area of the specimen. Images of each area (image tiles) are recorded when the stage is stationary, after waiting long enough for vibrations from the moving stage to dissipate, and using an exposure time that is sufficient to record the fluorescence images. These image tiles can be butted together, or overlapped and stitched using computer stitching algorithms, to form one image of the entire specimen.
When tiling microscopes are used for fluorescence imaging, the areas surrounding each tile and the overlapping edges of adjacent tiles are exposed twice (and the corners four times) which can bleach some fluorophores. Exposure can be adjusted by varying the exposure time for each tile. If multiple fluorophores are imaged, a different exposure time is required for each, so each fluorophore requires a separate image at each tile position. Multiple exposure of the specimen for imaging multiple fluorophores can also increase bleaching of the fluorophores.
A prior art strip-scanning microscope for fluorescence imaging is shown in FIG. 2. A tissue specimen 202 (or other specimen to be imaged) mounted on microscope slide 201 is illuminated from above by illumination source 203. In fluorescence imaging the illumination source is usually mounted above the specimen (epifluorescence) so that the intense illumination light that passes through the specimen is not mixed with the weaker fluorescence emission from the specimen, as it would be if the illumination source were below the specimen. Several different optical combinations can be used for epifluorescence illumination—including illumination light that is injected into the microscope tube between the microscope objective and the tube lens, using a dichroic beamsplitter to reflect it down through the microscope objective and onto the specimen. A narrow wavelength band for the illumination light is chosen to match the absorption peak of the fluorophore in use. Fluorescence emitted by the specimen is collected by infinity-corrected microscope objective 215 which is focused on the specimen by piezo positioner 220. Emission filter 225 is chosen to reject light at the illumination wavelength and to pass the emission band of the fluorophore in use. The microscope objective 215 and tube lens 230 form a real image of the specimen on TDI detector array 240. An image of the specimen is collected by moving the microscope slide at constant speed using motorized stage 200 in a direction perpendicular to the long dimension of TDI detector array 240, combining a sequence of equally-spaced, time-integrated line images from the array to construct an image of one strip across the specimen. Strips are then assembled to form a complete image of the specimen. When a CCD-based TDI array is used, each line image stored in memory is the result of integrating the charge generated in all of the previous lines of the array while the scan proceeds, and thus has both increased signal/noise and amplitude (due to increased exposure time) when compared to the result from a linear array detector. Exposure can be increased by increasing illumination intensity and/or by reducing scan speed. It is difficult to predict the best exposure before scanning. When multiple fluorophores are used on the same specimen, the usual imaging method is to choose illumination wavelengths to match one fluorophore, select the appropriate emission filter and scan time (speed) for the chosen fluorophore, and scan one strip in the image. Then the illumination wavelength band is adjusted to match the absorption band of the second fluorophore, a matching emission filter and scan speed are chosen, and that strip is scanned again. Additional fluorophores require the same steps to be repeated. Finally, this sequence is repeated for all strips in the final image. Some instruments use multiple TDI detector arrays to scan multiple fluorophores simultaneously, but because all fluorophores are scanned at the same scan speed, this usually results in a final image where one fluorophore is exposed correctly and the others are either under- or over-exposed. Exposure can be adjusted by changing the relative intensity of the excitation illumination for each fluorophore, which should be easy to do if LED illumination is used. When multiple illumination bands are used at the same time, the resulting image for each fluorophore may differ from that produced when only one illumination band is used at a time because of overlap of the multiple fluorophore excitation and emission bands, and because autofluorescence from the tissue itself may be excited by one of the illumination bands. Autofluorescence emission usually covers a wide spectrum and may cause a bright background in all of the images when multiple fluorophores are illuminated and imaged simultaneously.
A good description of strip scanning instruments, using either linear arrays or TDI arrays, is given in US Patent Application Publication # US2009/0141126 A1 (“Fully Automatic Rapid Microscope Slide Scanner”, by Dirk Soenksen).
When a strip-scanning instrument using either a linear array or TDI detector is used for fluorescence imaging, the fluorescence exposure is measured in advance, often by scanning the entire specimen and then using the resulting image to set scan speed and illumination intensity before making the final scan. When imaging specimens with multiple fluorophores, exposure for each fluorophore is measured separately. If exposure is not measured in advance, the result is often over- or under-exposed images.
Fluorescent specimens often emit a wide range of fluorescence intensity, which may require a wider dynamic range than the detection system can measure, even if the best exposure is set in advance. This is similar to the problem of photographing landscapes where the image brightness ranges from deep shadows through mid-tones to the bright sky with white clouds, and where detail must be preserved both in the shadows and in the clouds. In photography, HDR (High Dynamic Range) imaging is achieved by capturing several images of the same scene at different exposures, and merging them into the same image. Portions of the source images showing most detail are given most weighting in the merge process. A digital camera using this method is described in U.S. Pat. No. 5,828,793 (“Method and Apparatus for Production of Digital Images having Extended Dynamic Range”, by Steve Mann). A similar technique for scanning fluorescence microscope slides is described in U.S. Patent Application #US 2009/0238435 A1 (“Multi-Exposure Imaging for Automated Fluorescent Microscope Slide Scanning”, by Kevin Shields). In both of these descriptions, multiple source images with different exposures are combined to produce a single image in which detail is preserved in dark, mid-tone and bright areas of the image, but relative pixel intensities are not preserved across the image.
A different method for capturing a high dynamic range image of a specimen is described in U.S. Patent Application #US 2011/0134280 A1 “System and Method for Constructing High Dynamic Range Images” by Chou et al. In this method, a first image of the specimen is processed to generate illumination parameters that are then used to modulate the incident light intensity used for imaging various areas of the specimen, resulting in a composite image in which the illumination in different areas of the specimen has been varied to preserve detail in those areas of the image, but relative pixel intensities are not preserved across the image.
Definitions
For the purposes of this patent document, a “macroscopic specimen” (or “large microscope specimen”) is defined as one that is larger than the field of view of a compound optical microscope containing a microscope objective that has the same Numerical Aperture (NA) as that of the scanner described in this document.
For the purposes of this patent document, “fluorescence” includes but is not limited to single-photon excitation, two-photon and multiphoton fluorescence, spectrally-resolved fluorescence, and photoluminescence; and “specimen” includes but is not limited to tissue specimens, genetic microarrays, protein arrays, tissue arrays, cells and cell populations, biochips, arrays of biomolecules, plant and animal material, insects and semiconductor materials and devices. Specimens may be mounted on or contained in any kind of specimen holder. “Fluorophores” include synthetic fluorophores, fluorescent proteins, and quantum dots. “Autofluorescence” is fluorescence from endogenous molecules, like proteins in a tissue specimen.
For the purposes of this patent document, “exposure” means any combination of illumination intensity, scan speed (which when increased reduces dwell time for spot-scanning systems) or shutter speed (for linear detector arrays, e.g. CCD arrays or CMOS arrays).
Detector gain can be adjusted by changing PMT voltage and/or preamplifier gain in a spot-scanning instrument, or signal gain in an instrument using array detectors.
For the purposes of this patent document, TDI or Time Delay and Integration is defined as a method and detectors used for scanning moving objects, usually consisting of a CCD-based detector array in which charge is transferred from one line of pixels in the detector array to the next in synchronism with the motion across the detector array of a real image of the moving object. As the real image moves as a result of motion of the object, charge builds up as it is transferred from one line of pixels in the array to the next, and the result is charge integration similar to a longer exposure used with stationary imaging. When the image of one line on the object (and integrated charge) reaches the last row of the array, that line of pixels is read out. In operation the last line of pixels from the moving image is read out continuously, creating one row of pixels in the final mage at a time. One example of such a camera is the DALSA Piranha TDI camera.
“Contract” is defined as dynamic range contraction.
A “multispectral” image is one that contains data from several discrete and narrow detection bands. For example, when multiple fluorophores are imaged, the signal from each fluorophore is detected using a narrow-band detection filter. When these images are combined into a single image, it is a “multispectral” image. (No spectra are recorded, only data from a few narrow and discrete detection bands.)
When a spectrally-resolved detector is used to record the spectrum of fluorescence emission from a spot on a specimen, and the data from each spot on the specimen (each pixel position) are combined into an image, such an image is a “hyperspectral” image, and a fluorescence spectrum of each image pixel is measured at that pixel position on the specimen.
A “scan lens” is a colour-corrected and infinity-corrected lens with an external entrance pupil. A mirror scanner can be placed at the external entrance pupil position without requiring any intermediate optics between the mirror scanner and the scan lens. A “laser scan lens” is a scan lens designed for use with laser light sources, and is usually not colour-corrected.
For the purposes of this patent document, a sparse-pixel preview image is an image of at least part of a specimen comprised of equally-spaced pixels that have the same size and exposure as pixels in a final image of the same area of the specimen.